Methods, apparatus and data structures for providing a user interface which facilitates decision making

ABSTRACT

A user interface for facilitating a decision making process, such as planning a trip. A unified view of various types of information related to an event may be provided. The unified view may be presented in a simulated three-dimensional environment having different types of information depicted on different windows. Different types of information related to a common event may be visually linked. A window showing a articular type of information may be brought into a focus view for closer inspection by a user. Alternative decisions may be depicted to permit vagueness or uncertainty, particularly at early iterations in the decision making process.

§ 1. BACKGROUND OF THE INVENTION

§ 1.1 Field of the Invention

The present invention concerns a user interface which facilitates adecision making process, for example, for planning a trip. Inparticular, the present invention concerns a user interface thatvisually links related information rendered in multiple windows.Alternative choices may be visually depicted.

§ 1.2 Related Art

§ 1.2.1 The Problem of Decision Making and Planning

Over the past few decades, two trends have permitted computer users toaccess a wealth of data and information. The first trend was theexplosion in the amount of relatively inexpensive data storage means.The second trend was the increasing interconnection of computers, firstby local area networks (or “LANs”), and more recently by wide areanetworks (or “WANs”) which may be private, such as intranets forexample, or public, such as the Internet for example. Both of thesetrends are expected to continue.

Although computer users now can access a wealth of data and information,challenges in interpreting data and using data and information tocomplete tasks remain. Applications for statistically processing dataand for visually depicting (e.g., with pie charts, bar charts, graphs)data are widely available. Similarly, using data and information tocomplete relatively simple tasks is straight forward (e.g., determiningthe least expensive three or four bedroom home in Virginia using arelational data base).

However, many tasks involve making decisions based on a number offactors (or choices) and are subject to initial uncertainty orvagueness. Such decisions typically require fixing or limiting the valueof one or more factors (or choosing), interpreting the result, anditerating towards a desirable result. For example, in the context ofplanning a trip, factors may include (i) what one wants to do, (ii) whenone wants to do it, (iii) where one wants to go, (iv) how much one wantsto spend, etc. Choices in a group of factors may limit choices in thatgroup or another group. That is, options are narrowed as choices aremade or factors are fixed. For example, if a person defines a budget of$1000.00 for a vacation, and decides to spend $400.00 on airfare and$400.00 on hotel accommodations, and decides that they will need $150.00for meals, that person will not have the option of a $200.00 helicoptertour unless they increase their budget.

Again, in the context of planning a trip, such as a personal vacationfor example, a person in Virginia might want to see a WashingtonRedskins football game, visit an aquarium, visit Aunt Betty in Delaware,see the Statue of Liberty and see autumn foliage. Time constraintsshould be accounted for. For example, when will a user's work scheduleallow them to take a vacation, on what dates do the Washington Redskinsplay at home, when are aquariums and the Statue of Liberty open, whenwill Aunt Betty be home. Places may be determined. For example, do Iwant to see autumn foliage in Virginia or Vermont and do I want to visitthe public aquarium in Baltimore, Md. or Camden, N.J. Budgetary limitsmay also be checked. For example, are those Redskins tickets going tocost too much? As choices are made, other choices are affected. Forexample, the peak time for autumn foliage in Vermont is late September,but the peak time for autumn foliage in Virginia is mid October.Further, data and information related to each of the factors may be bestrepresented to a user in different ways. For example, time or datefactors may be best represented on a calendar, cost factors may be bestrepresented as a summed number, geographic factors may be bestrepresented on a map, etc.

§1.2.2 Known Decision Making User Interfaces and their PerceivedLimitations

The foregoing trip planning scenario illustrates the challenges involvedin performing or planning complex tasks having a number of variables.The articles: L. Tweedie, R. Spence, D. Williams, and R. Bhoghal, “TheAttribute Explorer”, Video Proceedings, CHI '94, Boston, Mass., ACMPress (1994), downloaded fromhttp://www.ee.ic.ac.uk/resarch/information/www/LisaDir/at tv.html onJul. 29, 1998 (hereafter referred to as “the Attribute Explorerarticle”) and H. Dawkes, L. Tweedie, and R. Spence, “VICKI—TheVIsualisation Construction KIt”, Advanced Visual Interfaces Conference,Gubbio, Italy (May 27-29), downloaded fromhttp://www.ee.ic.ac.uk/research/information/www/LisaDir/VICKI/VICKI.html on Jul. 29, 1998 (hereafter referred to as “the VICKIarticle”) discuss tools for exploring “problem space” using a twodimensional graphical user interface (or “GUI”). The GUIs discussed inthe Attribute Explorer and VICKI articles use sliders to permitparameters to be adjusted. Results are then fed back to the user viacolor coded histograms. More specifically, the Attribute Explorer andVICKI articles discuss manipulating factors (e.g., type, price, numberof bedrooms, and garden size) in a house purchase task or decision. Thevarious attributes are related via color.

Unfortunately, the GUIs discussed in the Explorer and VICKI articles aretwo dimensional and depict data related to various factors in the sameway. Therefore, it is believed that these GUIs are not as useful whenvarious types of data or information are best depicted in differentways.

The article, L. Tweedie, R. Spence, H. Dawkes, and H. Su, “The InfluenceExplorer—A Tool for Design”, downloaded fromhttp://www1.acm.org:81/sigchi . . . ceedings/videos/Tweedie/I t2txt.htmon Jul. 29, 1998 (hereafter referred to as “the Influence Explorerarticle”) discusses a similar GUI in which the relationships betweenperformance results and design parameters are depicted. Unfortunately,as with the GUIs discussed in the Attribute Explorer and VICKI articles,the GUI discussed in the Influence Explorer article is one dimensionaland depicts data related to various factors in the same way. Therefore,it is believed that this GUI is not as useful when various types of dataor information are best depicted in different ways.

The VISAGE product, from MAYA Design Group, Inc. of Pittsburgh, Pa.permits information objects to be depicted in various frames (e.g., ahierarchical outline, a bar chart, and a map). Using a hierarchicaloutline, a user can navigate through the information by “drilling down”(or segmenting aggregated data) and “rolling up” (or aggregatingsegmented data). Each of the frames has a script which governs theappearance of objects that the frame contains. Color is used tocoordinate the same object shown in various frames. Objects can bedragged and dropped within and between various frames. Although theVISAGE product depicts data in various ways, the same data (not merelyrelated information) is depicted in each of the frames. Furthermore, theVISAGE product depicts objects in frames in only one or two dimensions.The VISAGE product is discussed in the articles: S. Roth, M. Chuah, S.Keredjiev, J. Kolojujchick and P. Lucas, “Towards and InformationVisualization Workspace: Combining Multiple Means of Expression”,downloaded fromhttp:www.cs.cmu,edu/˜sage/HCI-journal-96/HCI-journal.html on Jul. 29,1998; J. Kolojujchick, S. Roth, and P. Lucas, “Information Appliancesand Tools in Visage”, IEEE Computer Graphics and Applications, pp. 32-41(July/August 1997); P. Lucas and S. Roth, “Exploring Information withVisage”, downloaded fromhttp://www.maya.com/visage/base/papers/mayaVid.htm on Jul. 29, 1998; andP. Lucas, C. Gomberg, and S. Roth, “Visage: Dynamic InformationExploration”, downloaded fromhttp://www.maya.com/visage/base/papers/mayaDemo.htm on Jul. 29, 1998.

None of the “decision making” user interfaces introduced above exploitthree-dimensional graphics. As discussed in § 1.2.2.1 below, some knownthree dimensional user interfaces are concerned with accessinginformation, but not with using information when making decisions.

§ 1.2.2.1 Known Three Dimensional User Interfaces to Information

User interfaces providing a three dimensional depiction of relatedinformation have been discussed. For example, the article: Robertson, etal., “The Next Generation GUIS: Information Visualization Using 3DInteractive Animation,” Communications of the ACM, Vol. 35, No. 4, pages57-71 (April 1993) (hereinafter referred to as “the InformationVisualizer article”) discusses various proposals for an “InformationWorkspace”. The Information Visualizer article discusses a perspectivewall which permits information, having some sort of linear relationshipor thread, to be presented in the relatively narrow aspect ratio of atypical video monitor. When an item is selected, the wall moves the itemto a center panel as if it were a sheet in a player piano reel. Itsintuitive three dimensional metaphor allows smooth transitions amongviews, thereby helping a user to perceive object constancy. Files may beclassified by their modification date. Although the perspective walltechnique lends itself to information having a linear (e.g., timeline)thread, it is less useful for other types of information, or forinformation in which a linear thread is unimportant. Furthermore, theperspective wall is not specifically designed for facilitating aniterative planning or decision making process.

Similarly, in the video, R. Kullberg, “Dynamic Timelines: Visualizingthe History of Photography”, CHI '96 Video Program, first 4:19, alsodiscussed in the thesis of the same title, downloaded fromhttp://robin.www.media.mit.ed/people/robin/thesis on Aug. 5, 1998(hereafter referred to as “the Dynamic Timelines thesis”), a threedimensional user interface to photographs is discussed. Unfortunately,the photographs have a linear, and more specifically, a temporal,thread-. Also, like the perspective wall, the dynamic timeline is notspecifically designed for facilitating an iterative planning or decisionmaking process.

§ 1.2.3 Unmet Needs

In view of the foregoing, there is an unmet need for a user interfacewhich facilitates planning or making decisions based on a number offactors (or choices). Since factors may be subject to initialuncertainty or vagueness, such a user interface should facilitate aniterative planning or decision making process, allowing the user to makepartial and/or vague decisions during the process. The results ofalternative decisions should be visualized. Relationships betweeninformation should be depicted. These depictions are not limited torelating the same information shown in different ways. The userinterface should permit a single display to include different windows ofdifferent types of related information such that a unified view of thetask is presented to a user. However, the user interface should permiteasy user interaction with any one of the different windows. Finally,intelligent help and access to information that may be needed or usefulto complete the task should be provided to the user.

§ 2. SUMMARY OF THE INVENTION

The present invention provides a graphical user interface in which anumber of windows or planes depict different types of relatedinformation. The consequences of changes to one type of information onone window are depicted in the other types of information on the otherwindows.

The present invention may further provide a graphical user interface inwhich the relationship between information in various windows isvisually depicted, for example using color and colored rays. The coloredrays may be translucent so that they do not occlude any information“behind” them.

The present invention may simulate a three dimensional environmentincluding different windows or planes having the different types ofinformation. The windows or planes may be related to one another assurfaces of a folded paper, or as facets of a geometric object that areconnected with one another. The present invention may permit any planeor window to be brought into a normal, head on, view (referred to as“focus view”) and depicted at a centered position on the display. Inthis regard, the present invention may permit the user to navigatebetween various windows, bringing any one of the windows into a focusview at any given time. The present invention may represent the windowsas surfaces or facets on a geometric object, such as a cube for example,and may employ animation when a user navigates from one window toanother. The present invention may use “world-in-miniature” (or “WIM”)techniques in addition to, or instead of the representation of thewindows on a geometric object, to facilitate navigation.

Finally, the present invention may facilitate the generation of queries,to be launched via an information browser, from words or phrases of freeform notes. The present invention may recognize relevant facts (e.g., inthe context of a trip planner, recognize dates, places, etc.) in oneinformation type to generate related information in another informationtype.

§ 3. BRIEF DESCRIPTION OF THE DRAWINGS

The file of this patent contains at least one drawing executed in color.Copies of this patent with color drawing(s) will be provided by thePatent and Trademark Office upon request and payment of the necessaryfee.

FIG. 1A is a personal computer on which the user interface of thepresent invention may be effected. FIG. 1B is a machine on which theuser interface of the present invention may be effected.

FIGS. 2A and 2B are displays of default views of a user interface inaccordance with an exemplary embodiment of the present invention. FIG.2C is a display of folders, such as that of FIGS. 2A and 2B, arranged ina box. FIG. 2D is a display of the box of folders of FIG. 2C on adesktop.

FIG. 3 is a display of a calendar window, in a focus view, of a userinterface in accordance with an exemplary embodiment of the presentinvention.

FIG. 4A is a high level diagram of a display, and FIG. 4B is an actualdisplay, of a map window, in a focus view, of a user interface inaccordance with an exemplary embodiment of the present invention.

FIG. 5A is a high level diagram of a display, and FIG. 5B is an actualdisplay, of a browser window, in a focus view, of a user interface inaccordance with an exemplary embodiment of the present invention.

FIG. 6A is a high level diagram of a display, and FIG. 6B is an actualdisplay, of a bulletin board window, in a focus view, of a userinterface in accordance with an exemplary embodiment of the presentinvention. FIG. 6C is a display of a note, in a preferred viewinglocation, the contents of which may be edited. FIG. 6D is a display of abrowser query launched based on the contents of a note. FIG. 6E is adisplay of a location on a map based on contents of a note.

FIG. 7 is a diagram of states and state transitions in an exemplaryembodiment of the present invention.

FIG. 8 is a diagram of processes which may be used to effect certainaspects of an exemplary embodiment of the present invention.

FIGS. 9 through 12 depict data structures which may be used in anexemplary embodiment of the present invention.

FIG. 13 is a flow diagram of an exemplary main routine process inaccordance with an exemplary methodology of the present invention.

FIG. 14 is a flow diagram of an exemplary process to get relatedinformation, which process may be called by the main routine process ofFIG. 13.

FIG. 15 is a flow diagram of an exemplary process to update viewingplane states, which process may be called by the main routine process ofFIG. 13.

FIG. 16 is a flow diagram of an exemplary process to manage a bulletinboard, which process may be called by the process of FIG. 15.

FIG. 17 is a flow diagram of an exemplary process to manage a map, whichprocess may be called by the process of FIG. 15.

FIG. 18 is a flow diagram of an exemplary process to manage a calendar,which process may be called by the process of FIG. 15.

FIG. 19 is a flow diagram of an exemplary process to generate visuallinks, which process may be called by the main routine process of FIG.13.

FIG. 20 is a flow diagram of an exemplary process to generate a focusview, which process may be called by the main routine process of FIG.13.

FIGS. 21A through 21C are displays depicting alternative map andcalendar windows.

FIG. 22 is an alternative default display of a user interface inaccordance with the present invention.

FIG. 23A is a display depicting an alternative calendar window. FIG. 23Bis a table of graphical representations which may be used in thealternative calendar window of FIG. 23A.

FIG. 24 is a display depicting an alternative map window.

FIG. 25 is a display depicting alternative combined calendar and mapwindows.

FIGS. 26 and 27 are displays depicting alternative combined calendar,map, and event views.

FIG. 28 is a display which shows an alternative arrangement of windowson a geometric object.

FIG. 29 is an alternative display of a trip plan.

FIG. 30 is a display depicting alternative calendar and map views.

FIGS. 31A and 31B are views of a display depicting alternative event andmap views.

FIG. 32 is a display depicting alternative map and calendar views.

FIGS. 33A and 33B are views of a display having alternative map,calendar, and event views.

§ 4. DETAILED DESCRIPTION

The present invention concerns novel methods, apparatus and datastructures for providing a user interface. The following description ispresented to enable one skilled in the art to make and use theinvention, and is provided in the context of a particular application(i.e., travel planning) and its requirements. Various modifications tothe disclosed embodiment will be apparent to those skilled in the art,and the general principles set forth below may be applied to otherembodiments and applications. Thus, the present invention is notintended to be limited to the embodiments shown.

Features of, and functions which may be performed by, the presentinvention will first be described in § 4.1 below. Then, structures,methodologies, data structures and displays of exemplary embodiments ofthe present invention will be described in § 4.2 below.

§ 4.1 Functions which may be Performed by the Present Invention

Recall from § 1.2.3 above, that there exists a need for a userinterface, and in particular a graphical user interface, forfacilitating planning or making decisions based on a number of factors(or choices). Recall also that since factors may be subject to initialuncertainty or vagueness, such a user interface should facilitate aniterative planning or decision making process and should visually depictthe results of alternative decisions. The present invention meets thisneed by providing a graphical user interface in which a number ofwindows or planes depict different types of related information. Theconsequences of changes to one type of information on one window aredepicted in the other types of information on the other windows.

Recall also from § 1.2.3 above, that relationships between information,not necessarily the same information depicted in different ways, shouldbe depicted. The present invention meets this need by providing agraphical user interface in which the relationship between informationin various windows is visually depicted, for example using color andcolored rays. The colored rays may be translucent so that they do notocclude any information “behind” them.

Recall also from § 1.2.3 above, that the user interface should permit asingle display to have different windows of different types of relatedinformation such that a unified view of the task is presented to a user.The present invention meets this need by simulating a three dimensionalenvironment including different windows or planes having the differenttypes of information. The windows or planes may be related to oneanother as surfaces of a folded paper, or as facets of a geometricobject that are connected with one another. However, recall from § 1.2.3that the user interface should permit easy interaction with any one ofthe different displays. One way the present invention meets this need isby permitting any plane or window to be brought into a normal, head on,view (referred to as “focus view”) and depicted at a centered positionon the display. However, this is not the only way a user can interactwith the displays—in an exemplary user interface, all of the (rendered)windows are always active. In such an exemplary user interface, thefocus view is merely provided to aid interaction for those users thatprefer a larger, normal view. In this regard, the present invention maypermit the user to navigate between various windows, bringing any one ofthe windows into a focus view at any given time. The present inventionmay represent the windows as surfaces or facets on a geometric object,such as a cube for example, and may employ animation when a usernavigates from one window to another. The present invention may use“world-in-miniature” (or “WIM”) techniques in addition to, or instead ofthe representation of the windows on a geometric object, to facilitatenavigation.

Finally, recall from § 1.2.3 above, that intelligent help and access toinformation that may be needed in completing the task should be providedto the user. The present invention meets this need in two (2) ways.First, the present invention may generate queries, to be launched via aninformation browser, from words or phrases of free form notes. Second,the present invention may recognize relevant facts (e.g., in the contextof a trip planner, recognize dates, places, etc.) in one informationtype to generate related information in another information type.

Having described functions which may be performed by the presentinvention, structures, methodologies, data structures and displays whichmay be used by the present invention are now described in § 4.2.

§ 4.2 Structures, Methodologies, Data Structures, and Displays which maybe Used by the Present Invention

In the following, exemplary systems on which the present invention mayoperate are described in § 4.2.1, exemplary displays which may begenerated by the present invention are described in § 4.2.2, a statediagram which illustrates exemplary interactions between the userinterface and a user is described in § 4.2.3, exemplary processes anddata structures which may be used to effect certain aspects of thepresent invention are described in § 4.2.4, flow diagrams showing anoperation of an exemplary methodology of the present invention aredescribed in § 4.2.5, and alternative displays which may be generated bythe present invention are described in § 4.2.6.

§ 4.2.1 Exemplary Systems

FIG. 1A and the following discussion provide a brief, generaldescription of an exemplary apparatus in which at least some aspects ofthe present invention may be implemented. The present invention will bedescribed in the general context of computer-executable instructions,such as program modules, being executed by a personal computer. However,the methods of the present invention may be effected by other apparatus.Program modules may include routines, programs, objects, components,data structures, etc. that perform a task(s) or implement particularabstract data types. Moreover, those skilled in the art will appreciatethat at least some aspects of the present invention may be practicedwith other configurations, including hand-held devices, multiprocessorsystems, microprocessor-based or programmable consumer electronics,network computers, minicomputers, set top boxes, mainframe computers,and the like. At least some aspects of the present invention may also bepracticed in distributed computing environments where tasks areperformed by remote processing devices linked through a communicationsnetwork. In a distributed computing environment, program modules may belocated in local and/or remote memory storage devices.

Referring to FIG. 1A, an exemplary apparatus 100 for implementing atleast some aspects of the present invention includes a general purposecomputing device in the form of a conventional personal computer 120.The personal computer 120 may include a processing unit 121, a systemmemory 122, and a system bus 123 that couples various system components,including the system memory 122, to the processing unit 121. The systembus 123 may be any of several types of bus structures including a memorybus or memory controller, a peripheral bus, and a local bus using any ofa variety of bus architectures. The system memory may include read onlymemory (ROM) 124 and/or random access memory (RAM) 125. A basicinput/output system 126 (BIOS), containing basic routines that help totransfer information between elements within the personal computer 120,such as during start-up, may be stored in ROM 124. The personal computer120 may also include a hard disk drive 127 for reading from and writingto a hard disk, (not shown), a magnetic disk drive 128 for reading fromor writing to a (e.g., removable) magnetic disk 129, and an optical diskdrive 130 for reading from or writing to a removable (magneto) opticaldisk 131 such as a compact disk or other (magneto) optical media. Thehard disk drive 127, magnetic disk drive 128, and (magneto) optical diskdrive 130 may be coupled with the system bus 123 by a hard disk driveinterface 132, a magnetic disk drive interface 133, and a (magneto)optical drive interface 134, respectively. The drives and theirassociated storage media provide nonvolatile storage of machine readableinstructions, data structures, program modules and other data for thepersonal computer 120. Although the exemplary environment describedherein employs a hard disk, a removable magnetic disk 129 and aremovable optical disk 131, those skilled in the art will appreciatethat other types of storage media, such as magnetic cassettes, flashmemory cards, digital video disks, Bernoulli cartridges, random accessmemories (RAMs), read only memories (ROM), and the like, may be usedinstead of, or in addition to, the storage devices introduced above.

A number of program modules may be stored on the hard disk 123, magneticdisk 129, (magneto) optical disk 131, ROM 124 or RAM 125, such as anoperating system 135 (for example, Windows NT® 4.0, sold by MicrosoftCorporation of Redmond, Wash.), one or more application programs 136,other program modules 137 (such as ReActor infrastructure and MicrosoftInteractive Sound System, both from Microsoft Corporation of Redmond,Wash., for example), and/or program data 138 for example. A user mayenter commands and information into the personal computer 120 throughinput devices, such as a keyboard 140 and pointing device 142 forexample. Other input devices (not shown) such as a microphone, joystick,game pad, satellite dish, scanner, or the like may also be included.These and other input devices are often connected to the processing unit121 through a serial port interface 146 coupled to the system bus.However, input devices may be connected by other interfaces, such as aparallel port, a game port or a universal serial bus (USB). The videomonitor 147 or other type of display device may also be connected to thesystem bus 123 via an interface, such as a video adapter 148 forexample. The video adapter 148 may include a graphics accelerator (e.g.,Intense 3D Pro 1000 or Intense 3D Pro 2200 from Intergraph Corporationof Huntsville, Ala.). One or more speakers 162 may be connected to thesystem bus 123 via a sound card 161 (e.g., a wave table synthesizer suchas product number AWE64 Gold Card from Creative Labs of Milpitas,Calif.). In addition to the monitor 147 and speaker(s) 162, the personalcomputer 120 may include other peripheral output devices (not shown),such as a printer for example.

The personal computer 120 may operate in a networked environment whichdefines logical connections to one or more remote computers, such as aremote computer 149. The remote computer 149 may be another personalcomputer, a server, a router, a network PC, a peer device or othercommon network node, and may include many or all of the elementsdescribed above relative to the personal computer 120, although only amemory storage device 150 has been illustrated in FIG. 1A. The logicalconnections depicted in FIG. 1A include a local area network (LAN) 151and a wide area network (WAN) 152, an intranet and the Internet.

When used in a LAN, the personal computer 120 may be connected to theLAN 151 through a network interface adapter (or “NIC”) 153. When used ina WAN, such as the Internet, the personal computer 120 may include amodem 154 or other means for establishing communications over the widearea network 152. The modem 154, which may be internal or external, maybe connected to the system bus 123 via the serial port interface 146. Ina networked environment, at least some of the program modules depictedrelative to the personal computer 120 may be stored in the remote memorystorage device. The network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

FIG. 1B is a more general machine 100′ which may effect one or more ofthe processes discussed above. The machine 100′ basically includes aprocessor(s) 102, an input/output interface unit(s) 104, a storagedevice(s) 106, and a system bus or network 108 for facilitating data andcontrol communications among the coupled elements. The processor(s) 102may execute machine-executable instructions to effect one or moreaspects of the present invention. At least a portion of the machineexecutable instructions and data structures may be stored (temporarilyor more permanently) on the storage devices 106 and/or may be receivedfrom an external source via an input interface unit 104.

§ 4.2.2 Exemplary Displays

As discussed in § 4.1 above, present invention may provide a graphicaluser interface in which a number of windows or planes depict differenttypes of related information. The display 200 of an exemplary graphicaluser interface, for use in the context of planning a trip, is depictedin FIG. 2A. The display 200 includes a number of windows 210, 220, 230,and 240. A first window 210 represents a bulletin board upon which notes(none shown in this Figure)' containing things that a user wants to do(e.g., see a Redskins game, visit Aunt Bettie, see autumn foliage, seean aquarium, and see the Statue of Liberty) are posted. A second window220 is a map for depicting places (e.g., Wash., D.C., Aunt Betty's hometown in Delaware, Vermont or Virginia, Baltimore or Camden, N.Y. harbor,etc.) to be visited during the vacation. A third window 230 is aninformation browser, such as an Internet browser for example, forfinding information related to things that the user wants to do.Finally, a fourth window 240 is a calendar for depicting when variousplaces will be visited during the vacation. In this example, thecalendar permits three (3) alternative trips to be compared and the tripwill span two (2) weeks.

FIG. 2B is a display 200′ in which the windows 210, 220, 230 and 240depict relevant information. For example, the bulletin board window 210includes a number of notes 212 on which the user may jot down thingsthat they want to do on their trip. The contents of the notes 212 areunconstrained—they are free form textual inputs. This facilitatesinitial user vagueness or uncertainty. The map window 220 depicts a map,in which places 222 to be visited are highlighted, with a colored markericon for example. The colors of the highlighting may match variouscolors of related notes 212. The Internet browser window 230 includesinformation which may be related to the notes 212 on the bulletin boardwindow 220, the places 222 on the map window 220, and/or to dates 241 onthe calendar window 240. For example, as described in more detail below,information entered on a note 212 may be used to generate a query whichis launched via the Internet browser 230. Similarly, a place entered ona map and/or a date entered on a calendar may also be used to generate aquery. Any geographic information in a web page rendered on the Internetbrowser 230 may be visually linked with related information on the mapwindow 220, with translucent colored ray 223. The information on the mapmay be visually linked with related information on the calendar 240,with translucent colored ray 224. Finally, the calendar window 240includes three (3) alternative trip scenarios 242. Each of the scenariosmay include one or more segments 246. The segments typically have acorresponding note 212 on the bulletin board window 210 which has acorresponding color.

Note that each of the windows 210, 220, 230, and 240 can represent asheet of a folder. Referring to FIG. 2C, various folders 201corresponding to various trips (or other objects of a decision makingprocess) may be stored in a box representation 203. As shown in FIGS. 2Cand 2D, the box 203 may be arranged on the top surface of a deskrepresentation 205. The desk 205, box 203, and folder 201 metaphors maybe rendered when starting and exiting the user interface of the presentinvention. Alternatively, a new or existing trip file may be opened inthe user interface of the present invention and managed via a known filemanagement utility, such as File Manager of Windows NT® for example.

Recall that user interaction with any given window should be easy. Byclicking the maximize button 214, 224, 234, or 244 of any one of thewindows, that window is presented in a focus view. A user may thenreturn to the default display 200′ by clicking the minimize button 314,414, 514, or 614 of any of the windows then in the focus view, which isdescribed below. Users may act on windows in both the default and thefocus view; the focus view serves to provide extra screen space to aspecific window.

FIG. 3 is a display 300 which depicts the calendar window 240 in focusview. The displayed calendar window 240 allows three (3) alternativeplans 242 to be entered. Each of the three (3) alternative plans 242 hasa first week (among the top group of the alternatives) and a second week(among the bottom group of the alternatives). Trip intervals 246 may beautomatically generated from temporal information within a note 212 onthe bulletin board window 210 and/or from temporal information extractedfrom a file rendered on the browser window 230, or they may be manuallyadded by a user. Calendar information may also be provided from anetwork-based groupware application such as Microsoft Exchange™. When acursor 299 is positioned “on”, or relatively close to, an interval 246,“handles” for manipulating the interval may be presented. In this case,a first handle 243 may be used to drag the interval, in its entirety,forward (right) or backward (left) in time. Further handles 244 a and244 b may be used to move the start or end of the interval,respectively, forward (right) or backward (left) in time.

In general, when an interval 246 is moved to an area (time or date)occupied by another interval, it will “push” that other interval.However, an interval may be fixed in time, either automatically or inresponse to a user command, such that when another interval is movedinto its area (time), that other interval will be divided over the fixedinterval, or jump, in its entirety, to the other side of the fixedinterval. As shown in FIG. 3, a fixed interval 247 may be rendereddifferently from normal intervals 246.

If a minimize button 310 is activated by the user, the trip folder 201is folded up and returned to the box of trips 203. If themaximize/minimize toggle button 244 is activated, default display 200′is reentered. Finally, as shown in FIG. 3, in addition to the calendarwindow 240, a WIM tool 3456 is displayed. In response to a useractivating (e.g., clicking) any of the faces of the WIM tool 3456, acorresponding window is displayed. The WIM tool 3456, even if inactive,serves as a map to aid inter-window navigation by the user. Thus, as canbe appreciated, when the calendar view 300 is displayed in focus view,an downward flicking gesture will cause the map window 220 to bedisplayed in the focus view.

FIG. 4A is a high level diagram of a display 400, and FIG. 4B is anactual display 400′, of a map window 220 in a focus view. Locations ofevents are depicted with markers such as circles, as shown in FIGS. 4Aand 4B. These circles may be colored to match colors of correspondingnotes 212 on the bulletin board window 210 and intervals 246 or 247 onthe calendar window 240. These markers may be moved by the user, forexample using traditional drag-and-drop mouse gestures. Further, theuser may add or delete events from the map. In one implementation, whenan event is added to the map, a note 212 with the name of the locationis added to the bulletin board 210 and a query may be launched. Finally,the user can navigate the map via zoom-in, zoom-out, and scrollingcommands or command tools (not shown).

If a minimize button 410 is activated by the user, trip folder 201 isfolded up and returned to the box of trips 203. If the maximize/minimizetoggle button 424 is activated, the default display 200′ is reentered.In addition to the map window 220, a WIM tool 3456 is displayed. Inresponse to a user activating (e.g., clicking) any of the faces of theWIM tool 3456, a corresponding window is displayed. To reiterate, theWIM tool, even if inactive, aids inter-window navigation. Thus as can beappreciated, when the map window 220 is displayed in focus view400/400′, a right, left, or upward flicking gesture will cause thebrowser window 230, the bulletin board window 210, or the calendarwindow 240, respectively, to be displayed in the focus view.

FIG. 5A is a high level diagram of a display 500, and FIG. 5B is anactual display 500′, which depict the browser window 230 displayed infocus view. The browser window 230 renders a retrieved web page, such asan HTML web page for example. Places (e.g., addresses) 511 and/or timesor dates 512 may be extracted from the rendered web page and displayedon the map window 220 and/or the calendar window 240, respectively. If aminimize button 510 is activated by the user, trip folder 201 is foldedup and returned to the box of trips 203. If the maximize/minimize togglebutton 234 is activated, the default display 200′ is reentered. As wasthe case with the calendar window 240 and the map window 220, inaddition to the browser window 230, a WIM tool 3456 is displayed. Inresponse to a user activity (e.g., clicking) any of the faces of the WIMtool 3456, a corresponding window is displayed. When in the browser view500/500′, a left flicking gesture will cause the map window 220 to bedisplayed in the focus view.

FIG. 6A is a high level diagram of a display 600, and FIG. 6B is anactual display 600′, which depict the bulletin board window 210 in focusview. A note 212 can be opened in response to a user command, such asclicking on an empty portion of the bulletin board for example. A blanknote 212 will then be brought to a preferred viewing position (e.g., tothe right foreground) as shown in FIG. 6C. Text may then be entered ontothe note 212′. Similarly, an existing note may be edited by selecting(e.g., via a mouse click) the note to bring it to the preferred viewingposition. From the preferred viewing location, the default view 200′ canbe reentered by deselecting (e.g., by double-clicking or a flickinggesture) the note 212′. When the note is returned to the bulletin board210, the text may be used by the system to launch a query using thebrowser (See, e.g., FIG. 6D.), any dates may be indicated on thecalendar window 240, and/or any places or addresses may be indicted onthe map window 220 (See, e.g., FIG. 6E.).

The notes 212 can be repositioned on the bulletin board 210 using, forexample, traditional drag and-drop mouse operations.

Referring back to FIGS. 6A and 6B, if a minimize button 610 is activatedby the user, trip folder 201 is folded up and returned to the box oftrips 203. If the maximize/minimize toggle button 214 is activated, thedefault display 200′ is reentered. In addition to the bulletin boardwindow 210, a WIM tool 3456 is displayed. In response to a useractivating (e.g., clicking) any of the faces of the WIM tool 3456, acorresponding window is displayed. As can be appreciated, when in thebulletin board view 600/600′, a right flicking gesture will cause themap window 220 to be displayed in the focus view.

Recall that in each of the focus views shown in displays 300, 400/400′,500/500′, and 600/600′ of FIGS. 3, 4A, 4B, 5A, 5B, 6A, and 6B,respectively, a world-in-miniature (or “WIM”) tool 3456 is provided.Recall further that the WIM tool 3456 includes a miniaturerepresentation of the display 200′ and includes a miniature bulletinboard window representation 210″, a miniature map window representation220″, a miniature browser window representation 230″, and a miniaturecalendar window representation 240″. Clicking any of the miniaturewindows representations will bring the corresponding window into thefocus view. World-in-Miniature tools are discussed in the article, R.Stoakley, M. J. Conway, R. Pausch, “Virtual Reality on a WIM:Interactive Worlds in Miniature”, downloaded fromhttp://www.cs.cmu.edu/-stage3/publications/95/conferences/chi/paper.html.)on Jul/ 30, 1998 (referred to as “the WIM article”).

§ 4.2.3 Exemplary User-GUI Interaction (State Diagram)

FIG. 7 is diagram illustrating states and state transitions that may beused by the exemplary trip planner of the present invention. When in astand-by or default state 710, the present invention may render display200′. If a cursor, the position of which may be controlled by a userinput to a pointing device which may provide a two-dimensional input, ispositioned over an empty spot on the bulletin board window 210 and anote creation command, such as a mouse click for example, is entered, orif the cursor is positioned over a note 212 on the bulletin board window210 and a note edit command, such as a mouse click for example, isentered, the bulletin board update state 720 is entered. This state 720is discussed in more detail in § 4.2.5 below with reference to FIG. 16.In this state, the user may (a) create a new note or (b) edit anexisting note. In either case, the note 212 being created or edited isbrought to the foreground of the display 200′. (Recall FIG. 6C.) Thenote may correspond to events or themes of the vacation (e.g., visitAunt Betty, see autumn foliage, see Redskins game, visit an aquarium,see the Statue of Liberty). The contents of the note 212 may be used togenerate a query to be launched via an information browser, such as anInternet browser or a address list (or “contacts” found in the Outlook™program sold by Microsoft Corporation of Redmond, Wash.). A query basedon the contents of the note 212 may be launched when a note 212 iscommanded, using a mouse flickering gesture for example, to be placedback (or “posted”) on the bulletin board 210. The movement of the note212 from the foreground of the display 200 to the bulletin board window210 may be accompanied by an animation. For example, a note “See theStatue of Liberty” may uncover a home page of the Statue of Liberty or anote “Visit Aunt Betty” may uncover a contact file for Aunt Betty. Thequery may also include other information, such as a user profile forexample. The user profile may be based on explicitly entered informationand/or information inferred from past use of the application, or otherapplications, by the user. For example, if the user profile indicatesthat a user is an architecture buff, the note “Statue of Liberty” mayalso uncover architecturally significant buildings in downtownManhattan.

The information returned may include information relevant to one of thewindows. In the context of the trip planning application, suchinformation (which may be referred to as “factoids”) may include dates,places, addresses, proper names, etc. Thus, if, as shown in FIG. 7, theinformation uncovered in the query includes time or date factoids, thecalendar generation or update state 740 is entered. This state 740 isdiscussed in more detail in § 4.2.5 below with reference to FIG. 18. Forexample, if a contact file for Aunt Betty includes her birthday, thatdate will be reflected in the calendar window 240 or if the Statue ofLiberty home page has a date of an upcoming special event, the date ofthat event will be reflected in the calendar window 240.

Similarly, if the information uncovered in the query includes geographicor address factoids, the map generation or update state 730 is entered.This state 730 is discussed in more detail in § 4.2.5 below withreference to FIG. 17. For example, if a contact file for Aunt Bettyincludes her home address, the address, at an appropriate level (street,city, county, state) of detail will be reflected in the map window 220or if the Statue of Liberty home page has its location, that informationwill be reflected, at an appropriate level of detail, in the map window220.

Naturally, if the geographic scope of the trip is defined in the mapwindow 220 or the temporal scope of the trip is defined in the calendarwindow 240, place or time factoids that fall within the definedgeographic scope or temporal scope, respectively, will be consideredparticularly relevant.

Referring to the bulletin board update state 720, if the cursor isremoved from the bulletin board, the standby state 710 is reentered.Also, in the map generation or update state 730 and the calendargeneration or update state 740, after the factoid induced map 220 orcalendar 240 is generated, the standby state 710 is reentered 710.

The map generation or update state 730 may also be entered from thestandby state 710 when the cursor is positioned on the map window 220.The standby state 710 is reentered when the cursor is taken off of themap window 220. Similarly, the calendar generation or update state 740may also be entered from the standby state 710 when the cursor ispositioned on the calendar window 240. The standby state 710 isreentered when the cursor is taken off the calendar window 240.

Also, from the standby state, if the map window 220, browser window 230,calendar window 240, or bulletin board window 210 is maximized, then themap focus view state 750 (Recall FIGS. 4A and 4B.), the browser focusview state 760 (Recall FIGS. 5A and 5B.) the calendar focus view state770 (Recall FIG. 3.), or the bulletin board focus view state 780 (RecallFIGS. 6A and 6B.), respectively, is entered into. As mentioned above,the standby state is reentered from any of the focus view states whenthe window then in the focus view state is minimized (Recall buttons310, 410, 510, and 610.).

As discussed above, each of the windows may be thought of as a surfaceof a geometric object, in this case, as four (4) sides of a cube forexample. Therefore, by rotating the cube, the window in a focus view canbe changed. For example, referring to FIGS. 2A, 2B, and 7, from the mapfocus view state 750, a right “flicking” (e.g., holding a mouse buttonbriefly while moving the mouse to the right) gesture causes entry intothe bulletin board focus view state 780, a left flicking gesture causesentry into the information browser focus view state 760, and a downwardflicking gesture causes entry into the calendar focus view state 770.Similarly, from the information browser focus view state 760, a rightflicking gesture causes entry into the map focus view state 750. Fromthe bulletin board focus view state 780, a left flicking gesture causesentry into the map focus view state 750. Finally, from the calendarfocus view state 770, an upward flicking gesture causes entry into themap focus view state 750. Transitions from one focus view state toanother may be accompanied by a short rotation animation and associatedaudio to reinforce the cube metaphor. As discussed above, though notshown in FIG. 7 for purposes of clarity, various focus view states maybe entered using a world-in-miniature (or “WIM”) tool 3456. Further, asdiscussed above, the WIM tool 3456, even if inactive, can serve as aninter-window navigation aide.

Having described exemplary displays and states which may be used by thepresent invention, exemplary processes and data structures are nowdescribed in § 4.2.4 below with reference to FIGS. 8 through 12.

§ 4.2.4 Exemplary Processes and Data Structures

§ 4.2.4.1 Exemplary Processes

data which may be used to effect, at least certain aspects of, the userinterface of the present invention. Basically, the processing by thepresent invention may be thought of as a sequence of cycles. In eachcycle, user inputs are accepted, states are determined, windows areupdated, if necessary, based on such user inputs, and a display isrendered (and audio is output) based on the state and the contents ofthe windows. Referring to FIG. 8, user inputs are managed by an inputmanagement process (or more generally, “an input manager”) 820.Referring back to FIG. 1A, this process may be carried out by thepointer 142, the serial port interface 146, and program modules 137. Anyuser inputs may be provided to an input command queue 822 for buffering.The queued commands are then read by an interface management process810.

The interface management process 810 will maintain a present location ofa cursor. The information browser view management process (or moregenerally, “an information browser manager”) 852, the bulletin boardview management process (or more generally, “a bulletin board manager”)854, the map view management process (or more generally, “a mapmanager”) 856, and the calendar view management process (or moregenerally, “a calendar manager”) 858 manage the browser window 230, thebulletin board window 210, the map window 220, and the calendar window240, respectively. Each of these processes maintains status informationof its associated window. If any one of these windows is maximized, orif a window representation in a world-in-miniature (or “WIM”) tool 3456is selected, the focus view generation process (or more generally, “afocus view generator”) 870 is invoked. Consequently, the outputgeneration process (or more generally, “an output generator”) 830 willgenerate a display with the appropriate window in the focus view.

If the window in the focus view is changed, for example by a flickinggesture or by selecting another window in the WIM tool 3456, then thefocus view update process (or more generally, “a focus view updatefacility”) 880 is invoked. The change from one window in the focus viewto another may be accompanied by a stored animation 882 and a sound cue884. The output generation process 830 generates a video display of thisanimation.

If a user enters information into any of the windows, such as the entryof a note of the bulletin board window 210 for example, the implicitquery process (or more generally, “a implicit query facility”) 840 isinvoked. The query may use the information browser, such as an Internetbrowser or address file for example, to facilitate the query. (Anexemplary implicit query method is disclosed in U.S. patent applicationSer. No. 09/152,713, entitled “Methods, Apparatus and Data Structuresfor Providing a User Interface which Exploits Spatial Memory in ThreeDimensions, to Objects, and which Visually Groups Matching Objects,”filed on Sep. 14, 1998, and incorporated herein by reference.) Theresults of the query are then submitted to a factoids extraction process(or more generally, “a factoids extractor”) 845. If any times, dates, orplaces are returned, the calendar window 240 or map window 220,respectively, are updated based on such factoid information.

Having described processes which may be used by the present invention,exemplary data structures which may be used by the present invention aredescribed in §4.2.4.2 below.

§4.2.4.2 Exemplary Data Structures

FIGS. 9 through 12 illustrate exemplary data structures which may beused by the present invention. Referring first to FIG. 9, recall thatnotes 212 can be entered or edited and posted on the bulletin boardwindow 210. The data structure 900 includes records 910. Each of therecords 910 corresponds to a note 212 and may include a field 912containing a value for uniquely identifying the note 212, a field 914containing a location of the note 212, a field 916 for indicating astate (e.g., active or inactive) of the note 212, and a field 918 forstoring content (e.g., text) of the note 212.

Referring to FIG. 10, recall that the calendar window 240 may include anumber of alternative trip plans. Each of the trip plans includes one ormore events (e.g., see a Redskins game, visit Aunt Betty, visit anaquarium, see autumn foliage, see the Statue of Liberty). The datastructure 1000 includes records 1010. Each of the records 1010corresponds to a trip plan and may include a field 1012 for containing aunique identifier for identifying each of the alternative trip plans,fields 1014 for containing the start time of each of the events, andfields 1016 for containing the end time of each of the events.

Referring to FIG. 11, recall that entering information, such as flickinga note 212 onto the bulletin board window 210 for example, in one windowmay be used to generate a query for processing by the informationbrowser. The returned information 1100 includes records 1110corresponding to each piece (e.g., a web page or a contact card) ofreturned information. Each record 1110 may include a field 1112 forcontaining an identifier for the related information, a field 1114 forcontaining a related information rank, and a field 1116 for containing arelated information state (e.g., active or top-of-stack, inactive,next-in-stack, etc.). Therefore, if the contents of a note 212 enteredon the bulletin board window 210 are used to generate a query launchedvia an Internet browser, a number of web pages may be returned. The webpages may be identified by their URL (“Uniform Resource Locator”), berank ordered based on their perceived relevance, and be active and shownon the browser window 230 or inactive. Thus, a number of web pages (orother content being browsed) may be cycled through by the user.

Referring to FIGS. 2B and 12, recall that related information in thevarious windows are visually linked, for example, with translucentcolored rays. An event defines a group of related information. That is,a Statue of Liberty event may include a note 212 to visit the Statue ofLiberty on the bulletin board window 210, a marker at the location ofthe Statue of Liberty on the map window 220, a home page of the Statuteof Liberty on the information browser window 230, and a date of aspecial event at the Statue of Liberty on the calendar window 240. Asshown in FIG. 12, each event has a record 1210. Each record 1210 mayinclude a field 1212 for containing a temporal location of the event onthe calendar window 240, a field 1214 for containing a location of theevent on the map window 220, a field 1216 for containing a location ofthe event on the browser window 230, and a field 1218 for containing aunique color associated with the event. Actually, since the calendarwindow 240 may include a number of alternative trip plans, it may show agiven event at different (or the same) dates in different tripalternatives. Therefore, the field 1212 may include a location of anevent for each of the alternative trip plans.

§ 4.2.5 Exemplary Methodologies

Having described various displays, processes, and data structures above,methodologies for effecting at least some of the processes described in§ 4.2.4.1 above are now described with reference to FIGS. 13 through 20.

FIG. 13 is a flow diagram of a main routine 1300 which may be run ormanaged by the interface management process 810. First, as shown in step1302, a user input is accepted. (Recalluser input management process 820and input command queue 822.) Next, as shown in decision step 1304, itis determined whether or not the user has entered an “exit” (or “quit”)command. If so, the main routine 1300 is terminated via return node1306. If, on the other hand, no exit command was entered, processingcontinues at step 1308 where a state of an appropriate window isupdated, if necessary. (Recall states 720, 730, and 740 of FIG. 7.)Next, related information is gathered in step 1310. (Recall the implicitquery process 840 and the factoids extraction process 845 of FIG. 8.)Then, the states of the other windows are updated, if necessary, in step1312. (Recall transitions from state 720 to states 730 and 740 in FIG.7.)

In decision step 1316, it is determined whether a particular window isin the focus view. (Recall states 750, 760, 770, and 780 of FIG. 7, andFIGS. 3, 4, 5, and 6.) If so, as shown in step 1318, the particularwindow is displayed in the focus view. (Recall process 870 of FIG. 8 andstates 750, 760, 770, and 780 of FIG. 7.) Processing then returns tostep 1302. If, on the other hand, no particular window is in the focusview, as shown in step 1320, then all of the windows are displayed.(Recall FIGS. 2A and 2B and state 710 of FIG. 7.) Since all of thewindows have been updated and may include related information, suchrelated information is visually associated. In step 1324, visual linksbetween related information in various windows are generated. (Recallprocess 860 of FIG. 8.) Also recall from FIG. 12 that relatedinformation are associated with an event and that the locations of therelated information in a given event are stored in records 1210.Processing then returns to step 1302.

Recall from FIG. 13, that related information is obtained in step 1310.FIG. 14 is a flow diagram of an exemplary process 1310′ for obtainingrelated information. First, an implicit query process is invoked asshown in step 1410. (Recall process 840 of FIG. 8.) The query may beformed, at least in part, based on words entered on a note. Though notdescribed in detail here, the query may also be formed, at least inpart, based on a place entered on the map window 220 or a date enteredon the calendar window 240. Recall that information from a user profilemay also be used when generating the query. Then, as shown in step 1420,a factoids extraction process is invoked. (Recall process 845 of FIG.8.) That is, if the information returned from the query includes a timeor date, or a place such information is depicted on the calendar window240 or the map window 220, respectively. The process 1310′ is thenterminated via return node 1430.

Recall from FIG. 13 that the state of an appropriate window may beupdated at step 1308. FIG. 15 is a flow diagram of an exemplary process1308′ for updating an appropriate one of the windows. First, at decisionstep 1510, it is determined whether or not the cursor is on the bulletinboard window 210. If so, a bulletin board view manager is invoked(Recall process 854 of FIG. 8.) as shown in step 1515 and processingcontinues at decision step 1520. If not, processing simply continues atdecision step 1520. At decision step 1520, it is determined whether ornot the cursor is on the map window 220. If so, a map view manager isinvoked (Recall process 856 of FIG. 8.) as shown in step 1525 andprocessing continues at decision step 1530. If not, processing simplycontinues at decision step 1530. At decision step 1530, it is determinedwhether or not the cursor is on the calendar window 240. If so, acalendar view manager is invoked (Recall process 858 of FIG. 8.) asshown in step 1535 and processing continues at decision step 1540. Ifnot, processing simply continues at decision step 1540. Finally, atdecision step 1540, it is determined whether or not the cursor is on theinformation browser window 230. If so, an information browser viewmanager is invoked (Recall process 852 of FIG. 8.) as shown in step 1545and the process 1308′ is terminated via return node 1550. If not, theprocess 1308′ is simply terminated via return node 1550.

Recall from FIG. 15 that the bulletin board view manager (Recall process854 of FIG. 8) may be invoked at step 1515. FIG. 16 is a flow diagram ofan exemplary process 1515′ for managing the bulletin board. Basically,the bulletin board management process 1515′ permits (a) notes to becreated or edited, (b) queries to be generated and launched, and (c)notes to be repositioned.

First, regarding creating (or editing) notes, as shown in decision step1610 if the user generates a note creation input, such as a mouse clickwhen the cursor is located on a blank part of the bulletin board window210 for example, a note is created as shown in step 1612 and enteredtext is accepted as shown in step 1614. Processing then continues atdecision step 1615. Returning to decision step 1610, if the user doesnot generate a note creation input, processing branches to decision step1615. At decision step 1615, if an edit command, such as a mouse clickwhen the cursor is located on a note 212 for example, is entered, thenote 212 is edited based on (mouse and/or keyboard) entries as shown instep 1616, and processing continues at decision step 1620. If a doubleclick is entered when the cursor is on a note 212, that note will be“pulled” from the bulletin board 210 and moved to a preferred viewinglocation as shown in FIG. 6C. If a double click is entered when thecursor is on a note in the preferred viewing location, it will bereturned to the bulletin board 210 at the position that it formerlyoccupied. Otherwise, processing simply proceeds to decision step 1620. Anote 212 being created or edited may be rendered at the foreground ofthe display 200′. (Recall FIG. 6C.)

Now, regarding the generation and launching of queries, if a userlaunches a note-based query, for example by flicking the note 212towards the bulletin board window 210, a query, based, at least in part,on the contents of the note 212, is generated and launched. Morespecifically, in the context of an Internet browser, a query is run forhome pages on the Internet based on the contents of the note as shown instep 1630. Then, the rank ordered web pages (Recall FIG. 12) areretrieved as shown in step 1640. Next, any factoids, such as times ordates and places, are extracted from the retrieved web pages as shown instep 1650. Next, as shown in decision steps 1660 and 1670, if there areno dates and places, the process 1515′ is terminated via return node1680. If, on the other hand, there are any places extracted, a relevantmap is retrieved as shown in step 1662. If any dates are extracted, acalendar is generated on the calendar window based on such dates, or acalendar on the calendar window is updated, as shown in step 1672.

Returning to step 1620, if a query is not launched, processing continuesat decision step 1690. At decision step 1690, it is determined whetherthe user is commanding to move the note, such as with a mouse dragoperation for example. If not, the process 1515′ is terminated viareturn node 1680. If so, the location of the note on the bulletin boardis updated in step 1692. A note drag may be accompanied by a scrapingsound, the pitch, volume, and timbre of which may be varied based on thevelocity and direction of the note drag.

Recall from FIG. 15 that the map view manager (Recall process 856 ofFIG. 8) may be invoked at step 1525. FIG. 17 is a flow diagram of anexemplary process 1525′ for managing the map. Basically, the mapmanagement process 1525′ permits (a) the map to be navigated and (b)events to be relocated or deleted. First, as shown in step 1710, a mapis generated based on the query results. For example, if the note 212that generated and launched a query read “visit a public aquarium”, theBaltimore and Camden aquarium home pages may have been returned. Thelocations of these aquariums would be extracted and a map window 220,showing both Baltimore, Maryland and Camden, N.J. would be generated. Asshown in FIG. 4A, the map generated may include markers, such astranslucent colored circles for example, matching the color of the note,at or around Baltimore, Maryland and Camden, N.J. If the markers arerendered as translucent colored circles, the radius of these circles maybe changed by user input.

Map navigation is now described. At decision step 1720, it is determinedwhether or not the user is commanding a zoom in or zoom out. If not,processing continues at step 1730. If so, as shown in step 1725, suchzoom in or zoom out commands are effected. Processing continues atdecision step 1730. At decision step 1730, it is determined whether ornot the user is commanding a scroll east, west, north or south. If not,processing continues at decision step 1740. If so, as shown in step1735, such scroll east, west, north, or south commands are effected.Processing then continues at decision step 1740.

Event manipulation is now described. At decision step 1740 it isdetermined whether a location is deleted from the map or not. If so, thedeletion is effected as shown in step 1745 and processing continues atdecision step 1750. Otherwise processing simply continues at decisionstep 1750. For example, if a map displaying and highlighting bothBaltimore, Md. and Camden, N.J. is displayed, and the user decides thatthey want to visit the Baltimore aquarium but not the Camden aquarium,the user can delete the marker at Camden, N.J. on the map. At decisionstep 1750, it is determined whether not the user is commanding an eventto be moved, such as with a mouse drag for example, on the map or not.If so, such a move is effected in step 1755 and the process 1525′ isterminated via the return node 1760. Otherwise, the process 1525′ issimply terminated via the return node 1760. For example, if the user hasdeleted Camden but later decides to visit the Camden aquarium instead ofthe Baltimore aquarium, they can drag the marker from Baltimore toCamden. In an alternative embodiment, rather than having a single map inwhich the user can zoom and pan, a fixed set of maps may be used by themap window 220.

Recall from FIG. 15 that the calendar view manager (Recall process 858of FIG. 8) may be invoked at step 1535. FIG. 18 is a flow diagram of anexemplary process 1535′ for managing the calendar. Basically, thecalendar management process 1535′ permits (a) the events to be moved onthe calendar, (b) events to be deleted from the calendar, and (c) theduration of events to be lengthened or shortened on the calendar. First,as shown in decision step 1810, it is determined whether a giveninterval on the calendar is active or not, An interval may become activewhen the cursor is located, or hovers, over it. If not, the process1535′ is terminated via return node 1870. If so, processing continues todecision step 1820.

At decision step 1820, it is determined whether or not the user iscommanding to lengthen the active interval, for example by an outwardmouse drag when the cursor is at an edge of the interface. Morespecifically, when an interval is active, “handles” may be rendered atits left and right edges. (Recall handles 244 a and 244 b of FIG. 3.) Ifthe user lengthens the duration of the interval, using the edge handlesfor example, the interface is lengthened as shown in step 1825 andprocessing continues at decision step 1830. Otherwise, processing simplycontinues at decision step 1830. At decision step 1830, it is determinedwhether or not the user is commanding to shorten the active interval,for example by an inward mouse drag when the cursor is at an edge of theinterface. If so, the interface is shortened as shown in step 1835 andprocessing continues at decision step 1840. Otherwise, processing simplycontinues at decision step 1840.

At decision step 1840, it is determined whether or not the user iscommanding the active interval to be deleted, for example by pressing adelete button or by dragging the interval to a trash icon. If so, theinterval is removed from the calendar as shown in step 1845 andprocessing continues at step 1850. Otherwise processing simply continuesat step 1850.

At decision step 1850, it is determined whether or not the user iscommanding to move the active interval, by a mouse drag when the cursoris at the center of the interval for example. More specifically, when aninterval is active, a drag “handle” may be rendered on it. (Recallhandle 243 of FIG. 3.) If so, the interval is moved as shown in step1855 and processing continues at decision step 1860. Otherwise,processing simply continues at decision step 1860.

At decision step 1860 it is determined whether or not the user enters adate. If so, the calendar is generated with the entered date depicted asshown in step 1862 and a query, based on the entered date, may begenerated in step 1864, before the process 1535′ is terminated viareturn node 1870. Alternatively, or in addition, if a note (or otherpiece of information of an event) is active, a mouse drag input on anunoccupied dates of the calendar will generate an interval related tothe event. Otherwise, the process 1535′ is simply terminated via thereturn node 1870.

Recall the calendar window 240 may have a number of alternative tripplans 242. Therefore intervals may be moved or copied from onealternative to one or more other alternatives, and an interval deleted,lengthened or shortened in one alternative will not affect an intervalto the same event in one or more other alternatives. Note, however, thatwhen an interval is automatically generated, such as when a date factoidis found in information returned in response to a query, all of thecalendar alternatives may be updated to include the event. If aninterval is manually entered in one calendar alternative, copies may beprovided to the other calendar alternatives if the dates in question areotherwise unoccupied.

Recall from FIG. 13 that visual links (Recall process 860 of FIG. 8) maybe generated between windows at step 1314. FIG. 19 is a flow diagram ofan exemplary process 1314′ for generating such visual links. Basically,for each event, a visual link is generated between information in eachwindow related to the event. Recall from FIG. 12 that each event has acorresponding record 1210. Beginning at step 1910, since all events areto processed, an event count (E_CT) is initialized (set to 1). Next, asshown in step 1920, a visual link, such as a colored (which may matchthe color stored in field 1218) translucent ray for example, isgenerated from the event (marker) on the map window 220 (the location ofwhich is stored in field 1214) to the event on the calendar window 240′(the location of which is stored in field 1212). Since the calendarwindow 240 may include multiple alternatives 242 which include intervalsrelated to the event, the visual link is terminated at the relevantinterval in an active or selected one of the calendar alternatives.Next, as shown in step 1930, a visual link, such as a coloredtranslucent ray for example, is generated from the event (marker) on themap window 220 to the event on the information browser window (thelocation of which is stored in field 1216). Thereafter, the event count(E_CT) is incremented at step 1940, Next, as step 1950 it is determinedwhether all of the events have been processed (E_CT>MAX). If so, theprocess 1314′ exits via return node 1960. If not, processing continuesat step 1920 at which the next event is processed.

Recall from FIG. 13 that if a particular window is in the focus viewstate, then the window is displayed in focus view (Recall process 870 ofFIG. 8) at step 1318. FIG. 20 is a flow diagram of an exemplary process1318′ for generating such focus views. Recall that a particular windowcan be brought into focus view in one of three ways; (i) if a window ismaximized, (ii) if a WIM representation of the window is clicked, or(iii) if a flicking gesture is input when another window is in focusview. Accordingly, at decision step 2010 it is determined whether or nota window was maximized, for example by the user clicking its maximize(214, 224, 234, or 244) button. If so, that window is shown head on andenlarged (i.e., in the focus view) as shown in step 2015 (Recall FIGS.3, 4A, 4B, 5A, 5B, 6A, and 6B.) and processing continues to decisionstep 2020. Otherwise, processing simply proceeds directly to decisionstep 2020.

At decision step 2020, it is determined whether the user selected arepresentation of a window in the WIM tool 3456. If so, that window isshown in the focus view as shown in step 2025 and processing continuesto decision step 2030. Otherwise, processing simply continues directlyto decision step 2030.

Finally, at decision step 2030, it is determined whether a flicking (up,down, left or right) gesture was input the by user. If so, the windowabove, below, to the left of, or to the right of the previous window infocus view is brought into focus view and the process 1318′ isterminated via return node 2040. (Recall focus view update process 880of FIG. 8.) The transition may be accompanied by a transition animation.Otherwise, the process 1318′ is simply terminated directly via returnnode 2040.

Having described the methodologies that may be used by the presentinvention, alternative displays are now introduced in § 4.2.6 below.

§ 4.2.6 Alternative Displays and Alternative Windows

FIG. 21A is portion of an alternative display 2100 in which thelocations 2122 of events on a map window 2120 are associated with theintervals related to the events on a calendar window 2140 by means oftranslucent arcs 2150. Note that the calendar window 2140 may use a“perspective wall” metaphor as discussed § 1.2.2.1 above. A thumb wheeltool 2162 is used to select various alternative trip plans. As shown,bar meters enable a user to quickly discern a trip alternative's cost2164, fun factor 2166 (for example, how many desired events willactually be accomplished by a given trip plan alternative) and time2168.

FIG. 21B is a similar display, but further conveys travel periods andrelates such travel periods to routes on a map. More specifically, atime period T_(SA) corresponds to a travel route from Seattle tolocation A, a time period T_(AB) corresponds to a travel route fromlocation A to location B, and a time period T_(BC) corresponds to atravel route from location B to location C. Note that the travel periodsmay be indicted with a car (or plane or train or boat) icon 2170. Notefurther that the travel periods on the calendar are black since each ofthese times will typically correspond to time between “events”, ratherthan an event itself. The locations of events on the map window areassociated with the intervals related to the events on a calendar windowby means of colored, perspective view ladders 2150. Finally.,buttons2184 and 2182 may be used to progress or backtrack, respectively,through a trip. FIG. 21C is a display 2100″ similar to that 2100′ ofFIG. 21B but which associates the locations of events on the map windowwith the intervals related to the events of the calendar window by meansof colored, translucent rays 2150′.

FIG. 22 is another alternative display having a map window 2220, acalendar time line 2240, and information 2230 in the form of web pages.The information may represent the desired events. The pages ofinformation are arranged on the calendar time line 2240 and are visuallylinked, via lines 2150, to markers 2222 on the map window 2220. Thedepth of each page may be used to represent the duration of the event,depicted on the web page, on the timeline.

FIG. 23A is an alternative calendar window 2340 in which multiple tripalternatives 2341 are depicted. In this alternative calendar window2340, ambiguity or uncertainty with respect to time is indicated by“washed out” color on an event interval. Elements 2343 and 2345 depictevents with certain durations. Elements 2342 and 2345 depict travel“legs” (which may be temporally between events or destinations forexample) of known durations. Element 2346 depicts a discrete event tooccur at an uncertain time, while element 2347 depicts a discrete eventto occur at a known time. Finally, element 2348 depicts an event havinga known start time but an unknown end time, while element 2349 depictsan event having unknown start and end times.

FIG. 23B is a table 2380 of graphical representations that may be usedto depict common combinations of time-slot status and event,destination, or travel-leg status in the calendar of FIG. 23A.Generally, the upper visual representation represents the duration of anevent, destination, or travel leg, while the lower bar represents theavailable slot of time in the trip, within which an event, destination,or travel-leg may take place. More specifically, the columns of thetable 2380 represent a time (e.g., a date and time), and the rows of thetable 2380 represent duration.

Rows 2391 and 2394 include representations that depict events,destinations, or travel legs having fixed durations. Rows 2392 and 2395include representations that depict events, destinations, or travel legshaving uncertain durations. Row 2393 includes representations thatdepict events, destinations, or travel legs that occur almostinstantaneously, that is, having a duration of about zero (0).

Column 2381 includes representations that depict events, destinations,or travel legs having known start and end times. Column 2382 includesrepresentations that depict events, destinations, or travel legs havingknown start times but uncertain end times. Column 2383 includesrepresentations that depict events, destinations, or travel legs havinguncertain start times but known end times. Column 2384 includesrepresentations that depict events, destinations, or travel legs havinguncertain start and end times. Column 2385 includes representations,similar to those of column 2382, but in which the known start time hasbeen fixed. Column 2386 includes representations, similar to those ofcolumn 2383, but in which the known end time has been fixed.

FIG. 24 is an alternative display having a map window 2420 on whichroutes 2424 of alternative trip plans are depicted. The lines 2422 atthe markers are used to distinguish one alternative route from another.The duration of segments of the trip are indicated by the spacing of thelines 2422—the denser the display of the lines 2422, the longer theduration of the segment.

FIG. 25 is another alternative display having a facetted cylindricalcalendar (each of the facets of the facetted cylinder, not shown,contains one of various alternative trip plans) 2540 which may berotated to bring various trip plan alternatives to the fore. The mapwindow 2520 includes lines 2522 which indicate travel between events.

FIG. 26 is yet another alternative display having a map window 2620, acalendar time line 2640, and event representations 2610 (sphericalmarkers) Lines 2650 may visually link some of the events 2610 to thecalendar time line 2640 and the map window 2620. Some events 2610 arenot associated with the time-line 2640 or the map 2620. Note that event2610′ has three alternative 2660 dates—(i) Saturday, (ii) Sunday andMonday, or (iii) Monday and Tuesday.

FIG. 27 is still another alternative display having a map window 2720,events 2710, and a calendar window 2730. Again, lines 2750 visually linkan event 2710 with related date and location information. The projectionfrom each event 2710 to the time-line 2730 indicates when the eventoccurs.

FIG. 28 is an another alternative display. Rather than faces of a cube,the windows 2812 are shown as facets of a geometric object 2810. Aworld-in-miniature (or “WIM”) tool 2820 is provided to assist innavigating the facets of the geometric object 2810, particularly when awindow is in focus view.

FIG. 29 is yet another alternative display 2900 which depicts a firstperson view of a trip. Events 2910 are marked in the display. A user canprogress forward in space 2942 or back in space 2944 from one locationto another. Indications of event locations that are not immediatelyviewable in the visible portion of the map may be provided. Finally,indications of event locations that are “behind” the virtual locationand orientation of the user may be provided. Such indications allow theuser to easily shift their view to event locations that are notcurrently in the field-of-view. A calendar time line 2940 is alsoprovided.

FIG. 30 is still another alternative display in which events are shownas disks 3012 having map views on a face. A window 3040 may include acalendar time line. The thickness of the disks indicates the duration ofthe event. A route 3050, which may be flattened in a focus view,indicates a route from a destination of a first event to a destinationof a temporally adjacent event. In this alternative display, the routeis represented as a flexible ribbon 3050. The start of the flexibleribbon 3050 is attached to a start disk 3012 corresponding to a locationat the start of the trip. Similarly, the end of the flexible ribbon 3050is attached to an end disk 3012 corresponding to a location at the endof the trip. Thus, if the ribbon 3050 and disks 3012 were laid out on aflat surface, the alignment of each piece would correspond to itslocation on a traditional flat projection map.

FIGS. 31A and 31B are views of a display 3100 depicting alternativeevent and map views. More specifically, in this case, events aredepicted pictorially, rather than textually. The map 3110 includesmarkers 3120 related to locations of the events. A pictorialrepresentation 3130 of the event is displayed adjacent to an associatedmarker 3120. A route 3140 between events is also depicted. A user cannavigate through their trip, traveling along the route 3140.

FIG. 32 is a display 3200 depicting alternative map and calendar views.More specifically, areas 3220 centered on the locations of events on amap 3210 are brightened. A translucent rod 3240 connects the maplocation of each event to a corresponding time length box 3250. The timelength boxes 3250 are projected onto a plan 3257 as shadows 3255. Theshadows 3255 may be labeled with corresponding days of the week asshown.

Finally, FIGS. 33A and 33B are views of a display 3300 havingalternative map, calendar, and event views. As was the case with thedisplay 3200 of FIG. 32 in the display 3300, areas 3320 centered on thelocations of events on a map 3310 are brightened. (FIGS. 33A and 33B arealternatives that superimpose route indications on a map.) A translucentwall 3360 has a base which follows a route between the locations ofevents on the map 3310. A height of the wall 3360 corresponds to thelength of time into the trip. This height value may be continuous asshown or may be quantized to days, hours, minutes or seconds, forexample. The exact time is indicated by the segment's projection ontothe time-line surface 3257. The translucent wall is projected onto alongitude-calendar (or alternatively latitude-calendar) plane 3367 toform projection 3365. The longitude-calendar (or latitude-calendar)plane 3367 may have horizontal lines corresponding to days. Further,vertical lines, corresponding the longitude (or latitude) of thelocation of the events may be provided on the longitude-calendar (orlatitude-calendar) plane 3367. As can be appreciated, predominatelyeast-west travel is best projected onto a longitude-calendar plane 3367,while predominately north-south travel is best projected onto alatitude-calendar plane (now shown).

In this example, the events are represented pictorially rather thantextually. The pictorial representations 3330 of the events may bearranged on parallel projections from the longitudinal locations of theevents on the map.

§ 4.3 Conclusions

In view of the foregoing, the present invention provides a userinterface for facilitating a decision making process, such as planning atrip. The present invention provides a unified view of various types ofinformation related to an event. The unified view may be presented in asimulated three-dimensional environment. A window showing a particulartype of information may be brought into a focus view for closerinspection by a user. Alternative decisions may be depicted to permitvagueness or uncertainty, particularly at early iterations in thedecision making process.

Although the described embodiments concern a user interface for planninga trip, the present invention can be applied to other decision makingprocesses. Although the information related to an event were describedas free form textual notes, a map, a calendar, and an Internet browser,other types of information related to an event may be depicted. Forexample, pictorial representations of events may be used instead of, orin addition to the free form textual notes. (Recall FIGS. 31A, 31B, 33Aand 33B.) Also, a list of contacts, “to do” notes, scheduledappointments, etc., associated with an Outlook™ application or documentsassociated with a Word application (both from Microsoft Corporation ofRedmond, Wash.) can be browsed or searched rather than, or in additionto, web pages on the Internet. Further, although events were entered infree form textual notes and queries for related information werelaunched based on the contents of such notes, information can be enteredand queries launched based on information from any of the windows. Forexample, a user can enter a marker on the map, and a query can belaunched based on the location of the marker on the map. Thus, thepresent invention is not intended to be limited to the embodimentsdescribed above

1. A user interface that facilitates decision making, comprising: areception component that receives one or more expectational event; and arendering component that displays the one or more expectational event ona first facet of a polyhedral display object.
 2. The user interface ofclaim 1, further comprising one or more information sources that includedisparate information regarding the one or more expectational event. 3.The user interface of claim 2, the one or more information sourcesemployed to gather, based at least in part on the one or moreexpectational event, a first datum of information and a second datum ofinformation.
 4. The user interface of claim 3, the first datum ofinformation projected by the rendering component on a second facet ofthe polyhedral display object
 5. The user interface of claim 4, therendering component projects the second datum of information on a thirdfacet of the polyhedral display object.
 6. The user interface of claim5, further including a mapping component that generates at least onenon-occlusional colored ray to associate the first datum of information,the second datum of information and the one or more expectational event.7. The user interface of claim 6, the rendering component projects theat least one non-occlusional colored ray that associates the firstdatum, the second datum and the one or more expectational event intothree-dimensional space to connect the first, second and third facets ofthe polyhedral display object.
 8. The user interface of claim 5, thefirst facet, the second facet and the third facet of the polyhedraldisplay object focusable based at least in part on a user action.
 9. Theuser interface of claim 8, the user action comprises holding a mousebutton while moving a mouse in a cardinal direction.
 10. The userinterface of claim 3, the first datum of information and the seconddatum or information dynamically adjustable based on changes in the oneor more expectational event.
 11. The user interface of claim 10, thefirst datum of information dynamically changeable based on changes inthe second datum of information.
 12. The user interface of claim 10, thesecond datum of information dynamically changeable based at least inpart on changes in the first datum of information.
 13. The userinterface of claim 2, the one or more expectational result employed toform a query to search the one or more information sources.
 14. A methodthat facilitates decision making, comprising: receiving an inclinationalevent; and displaying the inclination event on a first plane of athree-dimensional display object rendered on a visual display.
 15. Themethod of claim 14, further comprising utilizing a plurality of datasources to retrieve a first item of information and a second item ofinformation relating to the inclination event.
 16. The method of claim15, further including projecting the first item of information and thesecond item of information onto a second plane of the three-dimensionaldisplay object, and a third plane of the three-dimensional displayobject respectively.
 17. The method of claim 15, further comprisingassociating a non-occluding colored ray with the first item ofinformation, the second item of information, and the inclinationalevent.
 18. The method of claim 17, further comprising projecting thenon-occluding colored ray into one or more three-dimensional spacewithin the three-dimensional display object to connect the first plane,the second plane and third plane of the three-dimensional object. 19.The method of claim 15, the inclinational event employed to generate aquery to search the plurality of data sources.
 20. A user interface thatfacilitates decision making, comprising: means for receiving anaspirational event; means for rendering a multi-dimensional object;means for obtaining one or more information associated with theaspirational event; means for projecting the aspirational event and theone or more information associated with the aspirational event onto aplurality of planes that comprise the multi-dimensional object; andmeans for associating a non-occluding visible ray with the aspirationalevent and the one or more information projected onto the plurality ofplanes that comprises the multi-dimensional object.